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Abstract: The field legality of expertise for job applicants in the field of vocational education is one of the strong 
considerations. This recognition can obtain through accredited professional institutions such as the Professional 
Certification Institute (PCI), which operates independently. The problem arises that PCI in the automotive sector 
consists of several providers (PCI-TO, PCI- INA, PCI- TOP, etc.), and different levels (the 1“ Party PCI, 2™! Party 
PCI, and 3" Party PCI) so the quality is not the same and requires the right solution. As a result, the automotive PCI 
requires a reference standard as a professional competency test model standard. This study aims to test and produce 
a professional competency test model in automotive clusters, namely engine tune-up injection system (ETU-IS), that 
meet the characteristics and can be applied. The development subjects consist of PCI-TOP informants, Indonesian 
Training Center in Bandung (BBLKI), and P4TK BOE/VEDC in Malang as automotive competency experts. 
Instrument validation consists of construction, validation and feasibility test of the instrument. Data analysis uses 
alpha coefficients and Inter-Rater statistics. The findings of the study are (1) the assessment of professional 
competency standard assessment on ETU-SI using a Competency-Based Assessment approach in the form of work 
instructions, including case studies, demonstrations, and interviews that meet the criteria; (2) the fulfillment of 
instruments related to competency dimensions in aspects of competency testing consisting of TS, TMS, CMS, EMS, 
and JRES; and (3) verification of the infrastructure and facilities as well as the competency test the ethical 
requirements such as impartiality and delivery of recommendations have appropriate decision. The new finding in 
this study is that appraiser's behavior is the key to success in the implementation of competency tests, so technical 
and methodological abilities require as professional competency examiners. 


Keywords: Competence test model, professional competence, engine tune-up, injection system, professional 
certification institute, vocational education. 


1. Introduction 

Vocational education into the concept of policy in improving the economy (Abuselidze & Beridze, 2019; Becker, 2019; 
Mustapha & Greenan, 2002; Pavlova, 2014), in many countries, including Indonesia. The implementation of vocational 
education reaps several vital problems that hinder acceleration. Among them, high unemployment is sourced from 
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vocational education graduates (Ulya, 2019). They have difficulty in finding work under their field of expertise. The 
statement contradicts the conceptual framework of vocational education and training that is preparing graduates who are 
ready to work (Brown et al., 2017; Rojewski, 2009) so, the analysis of the implementation of vocational education, 
including context, input, the process needs to be re-evaluated by looking at the output and outcome that has occurred. On 
the other hand, vocational education must be oriented and as close as possible to the industrial world (Retnowati et al., 
2018; Rintala & Nokelainen, 2019) so that graduates are absorbed. 

Similar problems arise in the implementation of vocational education globally and internationally, where UNESCO- 
UNEVOC (Hollander & Mar, 2009) has released several factors including access rights, facilities, numbers, planning 
and administration, curriculum, learning, qualifications and even teacher conditions, information and communication 
technology, budgets, testing and certification as well as with the work of industry partners. The statement covers the 
entire system approach and must be improved in all aspects involved. Narrower in the regional problems identified by 
SEAMEO VOCTECH in vocational education (Chinonso & M, 2017) including ICT, relevance, curriculum, assessment, 
research and development, access and equality, management and quality assurance, poverty reduction, lifelong learning, 
decent work and levels of education resistance (formal, non-formal, informal). Researchers classify problems with a 
systems approach into three main priority relevance, quality and equality (Logli, 2016). 

Forms of efforts that have been made by the Indonesian government include four sectors, namely international, 
regional, national and local improvement. Various cooperation and training to improve human resources have been 
carried out with several countries that have successfully held vocational training. The regional level, as a form of 
partnership with ASEAN, has been collaborating in the ASEAN Economic Community (AEC) and the ASEAN Socio- 
Cultural Community (ASCC) (Indrawan, 2017; Resources, 2018) since December 2015. While, at the national level in 
the form of policies includes the reform process, including the relevance of the curriculum, strengthening outputs based 
on needs, and training in Human Resources. At the local level, the president has issued instructions contained in 
Presidential Instruction No.9 of 2016 concerning "Revitalization of Vocational Schools", in which the development of 
vocational education is left entirely to the provincial level, namely the governor. However, the focus of human resources 
becomes important in strategies to improve the quality of vocational education. 

This concrete action were taken as a form of implementation of Presidential Instruction No. 9 of 2016, namely the 
strengthening of human resources of BNSP institutions through PCI with authority to conduct a national competency 
test. The development of competency tests refers to the Indonesian Qualifications Framework (IQF). At the vocational 
school level, the implementation of competency tests uses the scope of the First Party PCI. It only shows the level of 
implementing institution or organization, PCI 1‘ Party were established by the educational institution or training 
institution with the primary objective of certifying work competency for human resources according to certain limits. 
PCI The 2™ party has the authority to conduct competency testing on employees required by the supplier organization. 
3"4 party PCI has the power to do competency tests on institutions or associations in the national interest. 

Researchers argue, even though they differ in scope and authority. However, industry players have different views 
on prospective workers. Competency certificates recommended by third parties have stronger legitimacy than other 
certifications. On the other hand, 3"! Party PCI with its authority has the right to conduct tests on 1‘ PCI, and vice versa 
that First Party PCI does not have the same power. The researcher also analyses, the number of 3" party PCI is minimal 
compared to 2" party PCI and 1% Party PCI. The indicates that the guarantees recommended are trustworthy and 
accountable to the user. Whereas, the number of 1“ party PCI providers requires quality assurance through the holding 
of competency tests. As for the implementation of competency tests, silo assessments are internal assessors (competent 
teachers) to test assessments (independent students). This means that the impartiality of recommendations becomes 
ambiguous for accountability. The depth of essence and implementation techniques in meeting the competency principle 
based on unit tests is biased. Thus, the professional competency test model can be adopted in First Party PCI. This answers 
the fulfilment of work industry standards based on IQF competency standards. 

The ongoing competency so far is the Skill Competency Test (SCT) (Rais, 2018) also referred to as the expertise 
competency test (ECT) (Hartoyo et al., 2018). The application of SCT is based on the freedom and ability of vocational 
education providers in the automotive sector in choosing technology-based packages and standard packages 
(Frovihandika et al., 2020). The implementation of the SCT process involves vocational teachers and industry 
representatives together to assess the SCT process in the form of an assessment score. The concept of competency test 
scores does not reflect the truth of one's expertise. This is what makes SCT bias and formal legality not strong in industry 
recognition. Some weaknesses in SCT implementation arise, including budget and routine programs. Therefore SCT can 
develop its process by integrating competency tests on PCI for better quality. The researchers agreed that students who 
receive competent recognition by PCI have the same recognition even better than the SCT. This integration process is 
essential in improving the quality of vocational education graduates. 

Planning on developing a professional competency test model focuses on the Engine Tune-Up Injection System 
(ETU-IS) scheme. The main reason is the work carried out in this competency includes various systems that are complex 
in four-wheeled vehicles, including basic competence, engine systems and electrical systems. Under current 
developments, especially learning in vocational education and applying, to the world of work, that technology-based 
injection systems are an absolute requirement to be mastered by prospective workers. Thus, this competency is 
considered, necessary, especially in the use of technology, including the use of tools and data analysis of the condition 
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of the machine and its electrical system. The approach to the development of competency test models is based on 
professional standards, namely BNSP and PCI in the automotive field. The testing phase of the competency model 
includes setting goals, gathering evidence, comparing evidence with objectives, and decision information. 

Assessment in testing the model using the Competency-Based Assessment approach (Ahmad & Rofiq, 2020; Heo et 
al., 2018) is carried out by means of the test phase of the case study instruction, interview and demonstration. Case studies 
are provided to measure the analysis of knowledge of assessments in detecting the emergence of problems in ETU-IS. 
This is adjusted to the problems that often occur in the field and the world of work. Demonstration tests are carried out 
to measure practical ability according to test competency. The level of success in resolving cases in the form of work 
actions and standardized references that are adjusted as well as the development of other products is done by interviewing 
as from the depth of information. Taking evidence at an assessment that has not been complied with will be reconfirmed. 
This means that this concept can be done from any competency, but it must be done as a whole. One unit of competency 
that is incomplete has an impact on the decision that the assessment is recommended not competent yet. 

Previous research, regarding the competence of ETU-IS, was limited to the concept of competency required by 4- 
stroke motor learning materials (Hartanto et al., 2018), the compatibility of the certification scheme with the industrialized 
world (Santoso & Hassan, 2018; Woo et al., 2018) with PCI on Electronic Fuel Injection (EFI) and conventional 
(Frovihandika et al., 2020), and literacy capabilities in the field of engine tune-up (Arifin et al., 2020; M. Nurtanto et al., 
2020; Muhammad Nurtanto et al., 2020). Thus, the renewal of this research is the development of a professional 
competency test model by testing experts in their fields. Therefore, analysis can refer to PCI to maintain quality standards 
and competency certificates obtained have full accountability. 


Ze Methodology 


This study uses the Borg & Gall model approach that were developed (Gall et al., 2003, 2007). Development trials were 
carried out in Banten Province, Indonesia. The determination of place were based on (1) the availability of The 1‘ Party 
PCI as well as a place for competency testing, the 2"! Party PCI which is verified as being sufficient in the test scheme 
of the ETU-IS; (2) availability of The 3" Party PCI-TOP assessors with nationally licensed technical and methodological 
capabilities; (3) completeness of the test infrastructure on the ETU-IS and an upgrade of the competency of the 
Conventional Tune-Up; (4) organizing the 3"? Party PCI Engine Tune-Up grade competency test cluster, so that the 
development model can be trialled; and (5) Involvement of Assessor from PCI-TOP as The 1* Party PCI, BBLKI for 
Bandung as The 2™ Party PCI and PCI at one of the vocational schools in Cilegon, Indonesia as The 1“ Party PCI, so 
that assessment standards get full input. The research was carried out in 2019. 

The subject of development is the competency test participant on the ETU-IS, as many as 30 respondents consisting 
of 5 educator respondents, 9 respondents working in automotive companies; and 6 respondents from undergraduate 
students with appropriate scientific fields, while, external assessors were three assessors based on The 3 Party PCI 
recommendations with the following characteristics (Table 1): 


Table 1 — Characteristic of Experts 


Experts Expertise Description 

Experiences from experts as BBLK Banten Province instructors who are currently 
active in the Ministry of Manpower team. For 16 years, he has been active in education 
and training programs in training work centers. Licensed competence at the senior level 
with 40 competency test units. He also joined with the Association of Indonesian 
Vocational Educators (AIVE), some PCI in its development include PCI-TO and PCI- 
TOP at the national level, and others are PCI in the 1‘ and 2"4 Party. 

Expert experience as an instructor at the Indonesian Vocational Training Center in 
Bandung. He was active in developing PCI management, and several Toyota, Daihatsu 
Expert 2 —_ and other companies became partners in the education and training fields. He also has 
vocational training experience in the automotive sector in Germany. Development of 
PCI institutions includes the 3"! and 2"! party. 

Experts experience as educators in vocational education colleges for five years. 
Competencies in the field of automotive experience, one of which is the Engine Tune- 
Up in conventional systems and injection systems. Join the Association of Indonesian 
Vocational Educators (AIVE), and PCI, Indonesia. 


Expert1 


Expert 3 


Data obtained in the development of competency test models is done in two ways, namely testing instruments and 
developing instruments. The assessment instrument data is in the form of work instructions and extracting interview 
evidence that refers to the 3"! party IQF level. Competency experts conduct test instruments that have been developed, 
based on understanding agreements, especially strengthening when competency tests take place so that the understanding 
obtained can be uniform and the instruments can be relied upon. The developed instrument implemented theoretically 
and also empirically. Based on the initial testing process, the development process is applied. The construct specification 
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were done by factor analysis. The level of reliability is calculated using alpha coefficients (Ritter, 2010) and the reliability 
assessment of professional competency tests using (Hallgren, 2012; Tang et al., 2015) Kappa Cohen (k). The following 
formula is applied: 


r= mak Set) terete (1) 
pa ete (2) 
N-Yife 
rt = reliability coefficient kK = agreement level of assessment 
k = item numbers fo | = frequency of observations 
Si = item variant fe | =expected frequency 
S*t = total variant N =the number assessed items 


Sources: (Ritter, 2010) and (Hallgren, 2012) 


The lower limit of the reliability coefficient for good test indicators is 0.70. Next, a descriptive analysis of the test 
variables and recommendations from the results of the implementation is carried out. 


3. Result and Discussion 


3.1. Mapping the Test Assessment of Professional Competence 


The scheme of competency test in the tune-up engine injection 
system 


Case Study of 


= 

Z “Problem Sensors” 

= DEMONSTRATIONS Decision of 
1S) Recommendatioy 
~ . 

Fé Case Study of Observation Interview 

Z “Problem 

=! Components” 


Not Competence Yet 


Fig.1 — The Framework for Model for Professional Competency Assessment 


The competency test mapping is explained in Figure 1. It shows that the competency test were carried out with three test 
standards; a case study measuring HOTS-oriented knowledge or understanding, a demonstration testing the ability in the 
form of skills, and an interview were conducted to look deeply into the behaviour and implementation of the competency 
dimension. The implementation phase is offered to participants to choose which tests taken precedence. The competency 
test planning includes setting the engine with various cases of damage to the components and sensor functions. However, 
backup components and replacement sensors have been prepared if participants report information and procedures 
correctly. The competency test in a case study were conducted before the participants take a demonstration test. The aim 
is to avoid the risk of severe damage caused by participant negligence. 

Demonstration tests are carried out in all competency testing units. This distinguishes between case study tests and 
interview tests. The following is a list of competency test units on ETU-IS referring to IQF. 


Table 2 - Code and List of Competency Unit Tests 


Code Unit Competency Unit 
OTO.KR10.001.01 Maintaining components 
OTO.KR10.009.01 Reading and Understanding Engineering Images 
OTO.KR10.010.01 Using and maintaining the measurement tools 
OTO.KR10.016.01 Following occupational safety and health procedures 
OTO.KR10.017.01 | Using and Maintaining Equipment and Supplies at Work 
OTO.KR10.018.01 Communicating at Work 


AuKwnA|Z 
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No. Code Unit Competency Unit 
7 OTO.KR20.001.01 Maintaining the engine and its components 
8 OTO.KR20.010.01 Maintaining the cooling system and its components 
9 OTO.KR20.014.01 Maintaining the Petrol Fuel System 
10 OTO.KR20.020.01 Maintaining the Emission Control System 
11 OTO.KRS50.001.01 Testing, Maintaining, and Replacing ACCU 
12. OTO.KRS50.011.01 Maintaining the Ignition System 
13. OTO.KR50.012.01 | Maintaining Engine Management Engine (EMS) 


The automotive industry association together with the education and training center, the BBLKI in Bandung, and the 
assessors provided notes based on the competency, needs to be developed (Sern et al., 2019). Industry and association 
involvement has an impact on the needs that were adapted to conditions in the world of work. The results of the discussion 
obtained several concepts, as follows: 

i) Indonesian National Work Competency Standards (IQF) in the automotive sector, four-wheeled vehicle sub- 

sector no. Kep. 16/Men/VII/2004. 

ii) The scope consists of routine maintenance and engine repair, which include routine maintenance of the engine, 

electricity, and fuel system lines periodically by performing the ETU-IS. 

iii) The list of competency units compiled based on the needs of the ASBEKINDO organization and "Auto Service". 


There are two acknowledgements of this competency test, namely expertise in ETU-IS and engine tune-up 
conventional. Strengthening of the test unit in the injection system is OTO.KR20.20 (emission control system) required; 
and OTO.RR50.012 (maintaining EMS). While other competency units are found in conventional systems. 

i) The assessment carried out includes aspects of knowledge, skills and attitudes during the competency test carried 
out. The assessment is carried out in the form of a conversation consisting of case studies, demonstrations and 
interviews (Fig. 1.) 

ii) The emphasis on testing principles, namely rejecting impartiality and conflicts of interest to maintain the 
credibility of the competency certificate. 

iii) The extraction of evidence is carried out in general and is not limited to see the characteristics of certain 
producers and appraisers that are competently approved. 


On the other hand, there are some findings from the implementation of competency tests that lasted the last few years, 
namely: 

i) The application of vocational competency tests or SCT based on percentage scores and not competent 
recommendations. 

ii) Examiners or assessors do not have the ability or technical expertise or methodological competence. 

iii) Impartiality between examiners and assessments has taken place. 

iv) Limited human resources on PCI at one of the vocational schools in Cilegon, Indonesia in the Engine Tune-Up 
assessment cluster in the Injection System, while the results of the validation of the infrastructure adequacy at 
the place of competency testing have been met. 


A total of 13 units of competency tests must be carried out in several stages including the stages of reading, 
maintaining, checking, measuring, replacing, and even caring for the environment, tools, machines, results and decision 
making. If one unit is not fulfilled, then the "competent" recommendation is declared a failure. However, the convenience 
of 13 units, does not have to be implemented sequentially, but it must be integrated. The technological approach were 
seen from 2 additional test units of the competency test cluster in a conventional tune-up, namely emissions control and 
EMS (Telsey, 2020; Warju et al., 2018). Participants must be able to apply the tools as data retrieval from overcoming 
problems that occur. The use of test equipment in the form of a “gas analyzer” and “scan tool” is a multi-skill technology- 
based competency in which participants must understand how to work, set the instrument, test parameters, concentration 
analysis and sensor codes according to the type of vehicle. When the testing process is underway, and a fatal error has 
occurred, the examiner has the right to warn, that the actions of the participants are declared dangerous. At the same time, 
the participants are declared to stop their work and are declared not competent yet. This avoids further damage and severe 
risk. 


3.2. Dimensions of Professional Competence on ETU-IS 


Competency identification is based on IQF as a material for developing test instruments. The Assessors consider two 
pieces of evidence, namely, direct and indirect evidence. Direct evidence is in the form of the implementation of 
competency tests. The indirect evidence is based on supporting documents that refer to one's expertise in the form of 
certificates, that are recognized become a consideration in competency tests. Moreover, the measured dimension (Leutner 
et al., 2017; Norton, 2013) is also based on the adequacy of the participant's actions on five competency dimensions, 
including: 
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i) Task Skill (TS) 
ii) Task Management Skill (TMS) 
iii) Contingency Skill Management (CMS) 
iv) Environmental Management Skills (EMS); and 
v) Job and Skill Role Environment (JREs) 


These dimensions are instrumental in the implementation of competency tests (Russo, 2016). 


Table 3 - Competency Dimensions Applied in the Competency Test Unit 


No. Code Unit Instruction 1 Instruction 2. Demonstration Interview 

1 OTO.KR10.001.01 / / X/TS / 

2  OTO.KR10.009.01 / / X/TS / 

3 OTO.KR10.010.01 / / X/TS / 

4  OTO.KR10.016.01 / / X/TS / 

5 OTO.KR10.017.01 / / X/JREs X/EMS / 
JREs 

6 OTO.KR10.018.01 / / X/TS i 

7  OTO.KR20.001.01 / / X/TMS / 

8  OTO.KR20.010.01 / / xX /TMS / 

9 OTO.KR20.014.01 / / xX /TMS / 

10 OTO.KR20.020.01 X/EMS/JREs / xX /CMS X/CMS/ 
EMS 

11 OTO.KRS50.001.01 / / xX /TMS / 

12 OTO.KRS50.011.01 / / X/TMS / 

13. OTO.KRS50.012.01 / xX /CMS xX /CMS xX /CMS 


Noted: (X) is performed. 


The competency adequacy requirements consist of five dimensions that are applied to the competency test unit (Table 
3.). This concept is not a standard concept in instrument development. Changes possibly occur as needed. The assessor's 
main focus is on codification (X), but other competency elements may be asked. Generally, in the field, interviews 
develop according to the needs, but they must take into account time. The unit has not been answered in full, and assessors 


have the right to give instructions to get in-depth information, even ask more if it is deemed necessary. 


3.3. 


good". Test results are stated in the following Figure 2: 
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Professional Competency Assessment (PCA) Instrument Test Results 


The PCA in Tune-Up Engine were carried out through constructing tests by experts. Assessments of validity, reliability, 
feasibility and planning are carried out to meet the analysis of model development. A score scale consisting of 1-4 has 
been converted to a percentage, and the final result is compared with the ranking criteria between "very bad" and "very 
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(C) Assessment in Interviews (D) Average test results by Experts 


Fig. 2 - Results of Assessments and Average Rate Instrument of Three Experts 


The instrument test results displayed indicate that the assessment of experts in the area of expertise gained was above 
90.0%. This acquisition were interpreted that the results of the assessment test meet the criteria. All indicators in the 
category are “very good”. The results of the reliability analysis using the Cohen kappa coefficient (K) are shown in the 
following Table 4: 


Table 4- Coefficient (kK) between Restricted Test, Performance Assessment, and Oral Assessment 


Case Study Performance Interview 
Assessor Assessment Assessment Assessment 
1 2 3 1 2 3 1 2 3 
1 0.93 0.70 0.85 0.67 0.81 0.76 
2 0.86 0.82 1 
3 
Average = 0.82 Average = 0.78 Average = 0.85 


The reliability of the model developed shows that the average value obtained include (a) the coefficient (Kk) of the 
test in the case study of 0.82; (b) coefficient (K) performance appraisal of 0.78; and (c) coefficient (kK) interview 
assessment of 0.85. A reliability score of close to were stated as a strong relationship. The reliability coefficient value 
has very good reliability and meets the competency test requirements. 


3.4. Results of Implementation of Professional Competency Test 


The planning phase were carried out before the assessors carry out the assessment intending to equalize the understanding 
and critical points that must be met in the assessment. This recommendation is appropriate for all assessors to equalize 
their perceptions. The assessors explain the rules for carrying out the competency test and ask the participants’ agreement 
in choosing which assessment comes first. 20 people as participants, were involved in testing competencies from various 
backgrounds. Background comparison is a separate finding in the development of test instruments. More details have 
been described as follows: 


Table 5 - Results of The Implementation of the Professional Competency Test in Terms of The Codification and 
Identity of the Subject 


AGI Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
A.G2 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
A.G3 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
A.G4 Cc Cc NC Cc Cc Cc Cc Cc NC NC NC Cc NC 
A.G5 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
ALI Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
A.L2 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc NC NC NC 
A.L3 Cc Cc NC Cc Cc Cc Cc Cc NC NC Cc NC NC 
A.L4 Cc Cc Cc Cc Cc Cc Cc Cc Cc NC Cc Cc NC 
A.LS Cc Cc Cc Cc Cc Cc Cc Cc Cc NC Cc Cc NC 
A.L6 Cc Cc Cc Cc Cc Cc Cc Cc Cc NC Cc Cc Cc 
A.L7 Cc Cc Cc Cc Cc Cc Cc Cc NC Cc Cc Cc Cc 
A.L8 Cc Cc Cc Cc Cc Cc Cc Cc Cc NC Cc Cc NC 
A.L9 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
AS1 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
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Table 5 — (Cont.) 


A.S2 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
A.S3 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
A.S4 Cc Cc Cc Cc Cc Cc ¢C Cc Cc Cc Cc Cc Cc 
A.S5 Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc Cc 
A.S6 Cc NC Cc Cc Cc Cc Cc Cc NC NC Cc NC NC 
(%) 100 95 90 100 100 100 100 100 80 65 90 85 65 
Codific 01 (General Competency) 02 (Core Engine) 05 (Core Electrical) 


ations 01:01 09:01 ~— 10:01 16:01 «17:01 18:01 01:01 «10:01 «14:01 20:01. — 01:01 11:01 12:01 
Note: AG (Ac Works / have worked); AL (acc Students); US (acc lectures) 
C (competence); NC (Not competence yet) 


Researchers analyzed the results of competency tests that have been conducted obtained and several findings based 
on job categories and competency tests. Five participants with jobs as teachers participated in the competency test with 
80% success (5) were declared competent. Looking at the background of participants with recommendations that are not 
competent yet are teachers outside the field of expertise, but they have 2 years of the learning experience and have 
experienced failure in unit coding tests (OTO.KR10.010.01, OTO.KR20.014.01, OTO .KR20.020.01, OTO 
-KR50.012.01). The assessors made it clear that the participants were incorrect in explaining and using measuring 
instruments in their application of technology. The most incompetent recommendations are for participants in the student 
category. Most of them failed to verify the unit test code (OTO.KR10.010.01, OTO.KR20.014.01, OTO.KR20.020.01, 
OTO.KR50.012.01, and OTO.KR50.011.01). In understanding the participants during the testing process, the assessor 
stated that any work performed was less professional. The definition explains that a test participant must have strong self- 
assessment abilities (knowledge, skills, and attitudes) whereas, 83.3% of teachers was declared as "competent". In the 
preparation and readiness of the superior teacher, it is possible to be their routine habits. Based on the level of competency 
units consisting of general competencies, core engine and core electricity are the biggest failures in controlling emissions 
and EMS. Competency in this unit, there are more complex cognitive and psychomotor skills. 

Some analyzes submitted by assessors related to the implementation of the test and recommendations made by 
participants in the future, including 11 "competent" assessments and 9 "not competent yet" assessments. In the study case 
assessment, 55% was declared competent, while the performance appraisal by 45% showed competence and the 
assessment by interview was 60%. Some competency units have the same characteristics and often appear on "not 
competent yet " recommendations. This means that the test material in the competency unit has an interest level that must 
be mastered better as follows: 


i. Maintaining the Emission Control System by 35.0% is not resolved. Most test participants had difficulty in using 
a gas analyzer. The most important thing in this test is the level of concentration and explanation of the 
concentration produced by the gas analyzer. Another incident is that setting, the use of inappropriate equipment 
makes the results not optimal. 

ii. Maintaining and improving Engine Management System (EMS) by 35.0% have not been mastered. This is 
related to the computer system that reads engine conditions based on control through sensors so that changes 
that occur must be able to be detected. Case studies provided with problems several sensor functions outside 
normal conditions; this makes it difficult for test participants while the primary competence of the ETU-SI is 
the mastery of the capabilities of the EMS. Besides the troubleshooting given with lightly damaged payloads, it 
is also not appropriately resolved, for example, the loss of the fuse function, jumper code according to engine 
characteristics. 

iii. Maintaining the Gasoline Fuel System by 20.0% is not controlled. The main thing is the flow check of the 
gasoline fuel system. Furthermore this competency unit uses two types of understanding, namely the mechanical 
fuel system and the electrical fuel system. 


Other competency units are still within normal limits. Judging from the background of the test participants who 
experienced the biggest failures were students. This indicates that the habit of doing engine tune-up work is not maximal 
besides the injection system is not new, but its existence is limited and there are few experts in the field. 


3.5. Analysis of the Results of Professional Competence Effectiveness Assessment 


Overall the instrument testing process and the implementation of a professional competency test and the subsequent 
stages of the model effectiveness test include aspects of Valid, Reliable, Flexible, Fair, Objective, Systematic, and Useful. 
Expert results were presented in the following Figure 3: 
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Fig. 3 - Results of the Effectiveness of the Professional Competency Test Model on the Tune-Up Engine Schema in the Injection System 


Based on the results of expert evaluations presented in Figure 3, the instrument's feasibility. All instrument testing 
criteria have been met by a score of more than 85%. Figure 3 (a) describes the distribution by 3 experts and Figure 3 (b) 
describes the overall aspect. The highest test results obtained are the following aspects of an objective at 100%, fair 
aspects at 98%, flexible aspects at 95%, valid aspects at 93%, reliable aspects at 92%, and systematic aspects at 91%. 
Obtained one aspect by 85%, which is a valid aspect. All instruments meet the assessment requirements in the "very 
feasible or very good" category. 

The objective aspect obtained by the highest percentage of all aspects is 100%. This means that the instruments 
assessed by experts have the same tendency. If tested on other assessments, it will be close to the same. The objective 
strength lies in the observer or the expert level in line with the argument (Inayah et al, 2017) the level of subjectivity and 
objectivity is strongly influenced by the assessor. If all assessors give a high percentage and do not have a tendency then 
the assessment instrument is able to measure the objectivity of the assessment. The fair aspect obtained as a percentage 
of 98% shows that fairly, there is no partiality in the instrument's evaluation. Lam (1995) argues that fair is equality and 
removes bias. Thus, the results of the fair aspect assessment show that during the assessment all experts were equal or 
fair. The flexible aspect is 95%, according to the principle of competency assessment. Flexibility means that during the 
implementation of the competency the testing phase can be carried out in accordance with the conditions in the field, in 
accordance with the abilities chosen by the assessment. Competency assessment refers to Australia, where there are four 
assessment requirements that must be met, namely valid, reliable, flexible and fair (Harris et al, 1995). Whereas 
systematic and usable aspects measure the extent to which the structure of conformity of the instrument is easily 
understood and can be used as a reinforcing aspect in the development of the PCA model. The results of the overall PCA 
instrument development meet the assessment requirements and can be used for the same scope (ETU-IS) in general. 


—— Effectivnesss Practicality Average 


100% 
95% 
95% 95% 


9 
90% eae 
88% 
85% 
9 
80% 8080 
1 2 3 4 


Fig. 4 - Competency Assessment Model Test Results 


In terms of eligibility, the effectiveness and practicability elements were obtained on average 91% and 89%. 
Obtaining a score of close to 100% indicates that the relationship of eligibility is very good. The overall effectiveness 
and practicality of the competency test obtained are 90% with a very good category. All procedures have been carried 
out and met the criteria for developing a competency test model of the engine Tune-up injection system schema. The 
overall improvement statistics are shown in the following Table 6: 
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Table 6 - Competency Achievement Statistics 


Case Study Performance Interview 


Statistics Assessment Assessment Assessment 
Score scores scores 
Subject 20 20 20 
Mean 8.9 7.6 8.4 
Median 8.9 8.1 8.8 
Mode 9.2 8.2 8.6 
Variance 3.5 0.3 2.2 
Range 1 3.5 5.2 
Minimum 75 6.5 4.0 
Maximum 10.0 10.0 10.0 


Overall review that the highest level of success lies in the case study. The scores obtained in the case study tests 
show the mean and median values are superior. This is due to the fact that practice tests and interviews have complex 
understanding and depth in meeting work instruction indicators (Treuer & Reynolds, 2017). The researcher reinforces 
that most of the test participants did not master the emission control work unit and EMS, though the approach of the 
automotive field with technology becomes the most important part, especially the development of the vocational 
education curriculum (Ersoy & Kiiciik, 2010). Automotive vocational education graduates must be able to master 
technology-based competencies. 

The development of a professional competency test model with the CBA approach that has been applied fulfils the 
characteristics, including considerations from associations and industry, testing instruments by experts, making reliable 
evidence, making decisions and delivering the results of decisions to be important to consider. The assessor must master 
all aspects and elements of the competency test. Therefore, the technical and methodological requirements are the main 
requirements. Development of competency test models by fulfilling the criteria is relevant to industry needs and can be 
used in the assessment principles of PCI in various levels, especially in organizing vocational education. 


4. Conclusion 

The professional competency test model in the engine tune-up injection system scheme becomes a recommendation for 
PCI in producing quality vocational graduates who are able to meet the needs of the industrial world. Recommended 
results include: first, the development of the instrument has a clear framework and consists of competencies aspects of 
knowledge, skills and attitudes that can be done in the form of tests case studies, performance tests and interviews; second, 
the competency dimension is fully involved in the competency test unit consisting of TS, TMS, CMS, EMS and JREs; 
third, the results of the assessment instrument test conducted by automotive experts amounted to 93.3% with strong 
relationship reliability, implementation of competency tests fulfilling the criteria, and effectiveness analysis in the 
excellent category, evidenced by the aspect of the model's effectiveness by 89%, and practicality of 90%. All competency 
test activities have been documented and referred to in PCI in vocational education. The researchers recommend that this 
study produces a competency test with an excellent standardized IQF benchmark. 
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